1 link tagged with all of: inference + llm alignment + verbalized sampling + diversity
Click any tag below to further narrow down your results
Links
The paper shows that typicality bias in human preference data drives mode collapse in aligned LLMs, narrowing their output diversity. It introduces Verbalized Sampling, a prompt that asks the model to list multiple responses with their probabilities, and demonstrates significant diversity gains across writing, dialogue, QA, and synthetic data tasks without retraining.
- Mode collapse comes from "typicality bias" in human preference data—annotators favor typical/safe responses, so even a perfect reward model trained on that data would still collapse diversity.
- Verbalized Sampling (prompting the model to list multiple responses with probabilities) recovers pre-training diversity at inference time, no retraining needed.
- This boosts creative writing diversity 1.6–2.1x over direct prompting, with human judges rating outputs 25.7% better on average, and it also improves downstream math solver accuracy when used to generate training data—all without hurting factual accuracy or safety.